Proactive creation of image-based products

ABSTRACT

A method for proactively creating an image product includes storing a library of specification terms for image products by a computer system, receiving a command from a user, tokenizing the command into a plurality of tokens by the computer system, matching one or more of the tokens to the specification terms in the library to determine specification parameters for an image product by the computer system, automatically identifying images based on the specification parameters by the computer system, and automatically creating a design for the image product that incorporates at least some of the images identified based on the specification parameters.

TECHNICAL FIELD

This application relates to utilization of digital images, and morespecifically, to the design and creation of products based on digitalimages.

BACKGROUND OF THE INVENTION

In recent years, photography has been rapidly transformed from chemicalbased technologies to digital imaging technologies. Images captured bydigital cameras can be stored in computers and viewed on displaydevices. Users can also produce image prints based on the digitalimages. Such image prints can be generated locally using output devicessuch an inkjet printer or a dye sublimation printer or remotely by aphoto printing service provider. Other products that can be producedusing the digital images can include photo books, photo calendars, photomug, photo T-shirt, and so on. A photo book can include a cover page anda plurality of image pages each containing one or more images. Designinga photobook can include many iterative steps such as selecting suitableimages, selecting layout, selecting images for each page, selectingbackgrounds, picture frames, overall Style, add text, choose text font,and rearrange the pages, images and text, which can be quite timeconsuming. It is desirable to provide methods to allow users to designand produce photo albums in a time efficient manner.

SUMMARY OF THE INVENTION

The disclosed methods and systems can significant increase the awarenessof users in personalized image products such as photobooks. A user canalso be given a chance to learn and visualize what an image productmight look like before he or she realizes the possible design of animage product based on his or her images. The memories of the users canthus be better preserved in more colorful and tangible ways for theusers.

Furthermore, the disclosed systems and methods provide more convenientways for a user to obtain designs of image products such as photo books.Images in a group can be automatically arranged into an image productdesign without being asked by a user. The user is provided with anoption to review, to edit, and to request a physical copy of the imageproduct to be made according to the design. As a result, the time andeffort for a user to create and obtain an image product aresignificantly reduced.

In one general aspect, the present invention relates to a method forproactively creating an image product. The method includes storing alibrary of specification terms for image products by a computer system;receiving a command from a user; tokenizing the command into a pluralityof tokens by the computer system; matching one or more of the tokens tothe specification terms in the library to determine specificationparameters for an image product by the computer system; automaticallyidentifying images based on the specification parameters by the computersystem; and automatically creating, by the computer system, a design forthe image product that incorporates at least some of the imagesidentified based on the specification parameters.

Implementations of the system may include one or more of the following.The specification terms in the library can specify one or more imagesources from which the images are to be obtained, wherein one or more ofthe tokens can be matched to the specification terms in the library thatspecify the one or more image sources, wherein the step of automaticallyidentifying can include identifying images at the one or more imagesources. The specification terms in the library can specify one or morepersons, wherein one or more of the tokens are matched to thespecification terms in the library that specify one or more persons,wherein the step of automatically identifying can include identifyingimages that include the one or more persons in the images. Thespecification terms in the library can specify an occasion that theimages are associated with, wherein one or more of the tokens arematched to the specification terms in the library that specify anoccasion, wherein the step of automatically identifying can includeidentifying images that are associated with the occasion. Thespecification terms in the library can specify a time period in whichthe images are taken, wherein one or more of the tokens are matched toone or more specification terms in the library that specify a timeperiod in which the images are taken, wherein the step of automaticallyidentifying can include identifying images that are taken in the timeperiod. The specification terms in the library can specify one or moreimage product types, wherein one or more of the tokens are matched to atleast one specification term in the library that specifies an imageproduct type, wherein the design for the image product is automaticallycreated in the image product type. The method can further includeselecting a subset of the identified images, wherein a number of theidentified images in the subset is within a predetermined rangecompatible with the image product type. The specification terms in thelibrary can specify a style or a format for an image product, whereinone or more of the tokens are matched to at least one specification termin the library that specifies a style or a format for an image product,wherein the design for the image product is automatically created in thestyle or the format. The method can further include manufacturing aphysical manifestation of the image product based on the design for theimage product. The specification terms in the library can specify arecipient for an image product, wherein one or more of the tokens arematched to a specification term in the library that specifies arecipient for an image product, wherein the physical manifestation ofthe image product is sent to the recipient after the step ofmanufacturing. The method can further include ranking the identifiedimages using predetermined criteria; and selecting a subset of theidentified images based on a result of the ranking step, wherein thedesign for the image product can incorporate the subset of theidentified images. The method can further include after the step ofmatching, presenting a question to the user by the computer system;receiving an answer from the user; tokenizing the answer into aplurality of tokens by the computer system; and matching one or more ofthe tokens to the specification terms in the library to determineadditional specification parameters for an image product by the computersystem. The step of automatically creating can include: assigning atleast a portion of the identified images into a first page group and asecond page group each comprising one or more images; placing the firstpage group of one or more images in the first page of the image product;and placing the second page group of one or more images in the secondpage of the image product. The step of receiving a command can includereceiving a text from the user, wherein the text is tokenized by thecomputer system. The step of receiving a command can include: receivinga voice message from the user; and recognizing a text in the voicemessage, wherein the text is tokenized by the computer system. The stepof tokenizing can include parsing the command. The step of matching caninclude matching the one or more of the tokens to morphologicalvariations of the specification terms in the library to determinespecification parameters for an image product by the computer system.

In another general aspect, the present invention relates to a method forproactively creating a photobook. The method includes identifying agroup of images by a computer system; automatically creating a designfor an image product by the computer system without receiving a user'srequest for designing the image product if the number of images in thegroup is determined to be within a predetermined range, wherein theimage product incorporates one or more images in the group; presentingthe design of the image product to a user for preview; and receiving anorder from the user for a physical manifestation of the image productbased on the design.

Implementations of the system may include one or more of the following.The group of images can be received in an upload from a computer deviceto the computer system. The group of images can be formed in anelectronic share from a first user to a second user. The group of imagescan be clustered when they are selected for printing. The group ofimages can be clustered when the images are placed in an electronicfolder or an electronic album. The group of images can be tagged by acommon label. The method can further include automatically selecting aformat or a style for the design of the photobook before the step ofautomatically creating a design for a photobook by a computer system.The method can further include determining if the number of images inthe group is determined to be within a predetermined range, wherein thepredetermined range varies depending on if the group of the images isclustered in an upload from a computer device to the computer system, orwhen the images are selected for printing, or when the images are placedin an electronic folder or an electronic album, or when the images areplaced in an electronic folder or an electronic album, or the images aretagged by a common label. The image product comprises a plurality ofpages. The step of automatically creating a design for an image productcan further includes: assigning at least a portion of the plurality ofimages into a first page group and a second page group each comprisingone or more images; placing the first page group of one or more imagesin the first page of the image product; and placing the second pagegroup of one or more images in the second page of the image product. Thestep of automatically creating a design for an image product can furtherincludes: automatically selecting a first page layout from a library ofpage layouts, wherein the first page group of one or more images areplaced in the first page according to the first page layout; andautomatically selecting a second page layout from the library of pagelayouts, wherein the second page group of one or more images are placedin the second page according to the second page layout. The method canfurther include allowing the user to select the format or the style forthe design of the image product after the step of presenting the designof the image product. The method can further include making the physicalmanifestation of the image product in accordance to the design. The stepof presenting the design of the image product to a user for preview caninclude sending an email message to the user to present the design ofthe image product. The image product can include multiple pages thateach is configured to display one or more images in the group of images.The image product can include a single page that is configured todisplay one or more images in the group of images.

In another general aspect, the present invention relates to a method forproactively creating a photobook. The method includes identifying agroup of images by a computer system; automatically selecting a formatand a style for an image product by the computer system if the number ofimages in the group is determined to be within a predetermined range;automatically creating a design for the image product in the selectedformat and the style by the computer system without receiving a user'srequest for designing an image product using the group of images,wherein pages of the image product incorporates one or more images inthe group; presenting the design of the image product to a user forpreview; and receiving an order from the user for a physicalmanifestation of the image product based on the design.

In another general aspect, the present invention relates to a method forproactively creating a photobook. The method includes identifying agroup of images by a computer system; automatically creating a designfor an image product by the computer system without receiving a user'srequest for designing the image product incorporating one or more imagesin the group if the number of images in the group is determined to bewithin a predetermined range; presenting the first design of the imageproduct to a user for preview; detecting a change in the group ofimages; automatically modifying the first design of image product tocreate a second design for the image product in response to the changein the group of images; and presenting the second design of the imageproduct to a user for preview.

Implementations of the system may include one or more of the following.The change in the group of images can include addition to the group ofimages or removal of images from the group of images. The change in thegroup of images can include a change in an image property of the groupof images. The image property can include a tag, a keyword, a title of afolder where the images are placed, or metadata at least one of theimages.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The accompanying drawing, which are incorporated in and form a part ofthis specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention:

FIG. 1 is a block diagram of a system for imaging applicationscompatible with the present invention.

FIG. 2 shows a flow chart for proactively creating a photobook inaccordance to the present invention.

FIG. 3 is an exemplified user interface showing an image album having aplurality of images.

FIG. 4 is an exemplified user interface showing an image album having aplurality of images, and an indication that a photobook has beenproactively created.

FIGS. 5A and 5B illustrate an exemplified user interface for showing aproactively created photobook.

FIG. 6 shows a flow chart for proactively and automatically creating adesign for a photobook in accordance to the present invention.

FIG. 7 illustrates sorting and grouping of images by image capture timein accordance to an aspect of the present application.

FIG. 8 illustrates sorting and grouping of images by image color contentin accordance to another aspect of the present application.

FIG. 9 illustrates sorting and grouping of images by image-capturelocation in accordance to another aspect of the present application.

FIG. 10 shows a flow chart for proactively creating an image product inaccordance to some embodiments of the present invention.

FIG. 11 shows a flow chart for proactively creating an image product inaccordance to some embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the present invention, the term “photo album” refers to a book thatincludes one or two cover pages and one or more image pages. Each of theimage pages can include one or more images. The image pages can alsoinclude text or image caption. The image layout can also include abackground pattern. In the imaging industry, photo albums can includephoto books, scrapbooks, snap books, and any books form including boundimage pages.

A computer 100 illustrated in FIG. 1 represents an exemplified hardwaresetup for executing software that allows a user to perform tasks such ascommunicating with other computer users, accessing various computerresources, and viewing, creating, or otherwise manipulating electroniccontent, that is, any combination of text, images, movies, music orother sounds, animations, 3D virtual worlds, and links to other objects.The system includes various input/output (I/O) devices (mouse 103,keyboard 105, display 107) and a computer 100 having a central processorunit (CPU) 121, an I/O unit 117 and a memory 109 that stores data andvarious programs such as an operating system 111, and one or moreapplication programs 113. The computer 100 also typically includesnon-volatile memory 110 (e.g., flash RAM, a hard disk drive, and/or afloppy disk or other removable storage media) and a communications cardor device 123 (e.g., a modem or network adapter) for exchanging datawith a network 127 via a communications link 125 such as a cable modem,DSL service or wireless Internet connection. The computer 100 of FIG. 1also can be connected to various peripheral I/O devices. One of the morepopular of such peripheral devices is a digital camera 108. The digitalcamera 108 enables users to take pictures (i.e., images), which aresaved in memory within the digital camera 108 in a digital format. Aftertaking and storing the images, the user can connect the digital camera108 to a computer 100 in order to upload the digital images to thecomputer's disk drive or other non-volatile memory 110. The user canalso wirelessly transfer digital images in the digital camera to thecomputer 100. Once the digital images are uploaded to the computer 100,the user can erase the digital images from the memory of the digitalcamera 108 so that the user can take and store additional images usingthe digital camera 108. Typically, the digital camera 108 is connectedto the computer 100 only while the user is uploading images to thecomputer's disk drive or other non-volatile memory 110. Once the digitalimages are stored on the computer 100, a user can perform variousoperations on them. For example, an image viewer application can be usedto view the images. A photo editing application can also be used totouch-up or otherwise modify the images. The computer 100 cancommunicate via Internet 150 with a server 160 provided by a serviceprovider, such as Shutterfly, Inc. A user operating the computer 100 canupload or download images between the computer and the server 160. Theuser can also electronically share his or her images facilitated by theservice provider. The server 160 can be in communication with a database165 that is also operated by the service provider. The database 165 canstore account information, and product design, product creation andproduct order information in association with the user.

It should be noted that the computer 100 can exist in otherconfigurations from the example shown in FIG. 1 and described above. Thecomputer can include a mobile device such as a smart phone, a digitalassistant device, and a wireless enabled mobile computer such as alaptop computer or a tablet computer,

Although large numbers of images are frequently captured by digitalcameras, people often are not aware of the customized image productsthat are available from image service providers such as Shutterfly Inc.Customizable image products include photobooks, photo calendars, photogreeting cards, photo stationeries, photo mugs, photo T-shirt, and soon, which can provide significant enhanced ways for preserving people'streasured memories in addition to viewing images on electronic displays.

Additionally, significant obstacles exist even for people who are awareof the availability of customizable image products. The design of aphotobook, for example, can take significant amount of time and effort.A user has to select a large number of quality and emotionallysignificant photos, increasing from different sources, such as, a localcomputer or phone, at a data storage operated by the service provider,or at a third party social site (e.g. Facebook™, Instagram™ . . . ). Theuser needs to select a format, and a style for the photobook. The userneeds to design or select layout for each page including a cover page,sort images in different page groups, and place the images ontoindividual pages. The photobook design can take many iterative stepssuch as selecting suitable images, selecting layout, selecting imagesfor each page, add text, which can often take hours to complete.

Once the user approves a design for an image product, the image product(e.g. a photobook) can be made according to the design locally by aprinter 130 connected to the computer 100, or remotely by the imagingservice provider at a printing facility 170.

It should be understood that the presently disclosed systems and methodsare suitable for proactively creating different image products otherthan photobooks. The image products can include multiple pages (i.e.faces or views) that each can display one or more images, such as, photobooks, photo calendars, photo scrapbooks, photo snapbooks, photocalendars, photo albums, a multi-face photo card. The image products canalso include a single page or view for displaying one or more images,which can include, for example, photo greeting cards, photo calendars,holiday cards, photo mugs, photo T-shirts, photo aprons, single photopages, photo collage pages, a photo stationery, photo banners, photomugs, photo mouse pads, photo key-chains, photo collectors, and photocoasters etc.

In some embodiments, referring to FIG. 2, an image product can be madeavailable to a user in the following process. It should be noted thatthe process can include substeps within each step and other steps notshown in FIG. 2. Additionally, certain steps in the process may beoptional.

A group of images are identified by a computer system such as thecomputer 100 or the server 170 (step 210). The images can be received ina group in an upload from a computer device (e.g. the computer 100,FIG. 1) to the computer system (e.g. the server 170, FIG. 1), or from adigital camera to a computer (e.g. the computer 100, FIG. 1). The imagescan form a group when the images are shared in an electronic messagefrom a first user (e.g. operating the computer 100, FIG. 1) to a seconduser via a computer server (e.g. the server 170, FIG. 1). The images canbe identified when a group of images are selected for printing. Theimages can also be identified when the images are placed in anelectronic folder or an electronic album for an occasion or event, orsimply for collecting pictures of a theme or interest (which can becalled “a picture picker”). As shown in FIG. 3, a user interface 300(e.g. a web browser) displays an image album 310 that can have an albumname 315 such as “Graduation Celebration” and a plurality of images 320.The image album 310 can include several pages of images that a user cannavigate to and view by clicking arrow buttons 340, 350. Furthermore,images may be tagged by a user or the service provider by labeled suchas “Hawaiian Vacation”, “Soccer Team”, “Birthday Party”, “Favorite”,etc. A group of images can be identified when they are tagged by acommon label such as “Hawaiian Vacation”.

The computer system then determines if the number of images is within apredetermined range (step 220). For example, the threshold for thenumber of images in the group can be 20 (i.e. the predetermined range is20 or higher). If the images in the group are within the predeterminedrange, the computer system automatically selects a style and a formatfor an image product (step 230). For example, the styles of the imageproducts can include, romance, seasonal, everyday, vacation, travel,wedding, baby, kids, birthday, school memories, sports, Mother's day,holiday, etc., each of which can include more detailed styles. In theexample of photobooks, the style of a photobook can specify the theme ofthe photobook such as colors, the background themes, the patterns, thepicture frames, the text font, positions of the text, and images on theinner pages as well as covers of the photobook. The format of aphotobook can include the size e.g. 5″×7″, 8″×8″, 12″×12″ etc.), thepaper stock materials for the pages and the cover, the type of binding,and the finish of the photobook.

In some embodiments, the predetermined range can depend on how the groupof images is selected, whether they are selected for printing, or forelectronic share, or placed in a single electronic folder or album, orreceived in an image upload, or tagged by a same label. For example, thethreshold number can be 15 for the images that are selected forprinting, and 20 for the images that are selected for electronicsharing. The different threshold image numbers for different imageaggregation sources is based on the observation that users conductingdifferent image operations often have different degrees of interests inseeing the design and making a physical image product such as aphotobook. The above described flexibility in image threshold numberscan provide service best suitable to different segments of users.

In some embodiments, the predetermined range can depend on the type ofimage products. For example, a calendar book may require 12 differentimages for 12 months plus an extra image for the calendar cover. Sincethe calendar cover may use a redundant image from the months, thereneeds to be at least 12 images for a calendar book. So the minimumnumber of images for a 12-month calendar book is 12. Similarly, theminimum number of images for an 18-month calendar book is 18. On theother hand, the design of a photo mug can be proactively prepared whenthere is at least one image in the group. A single-page photo collagemay require a minimum of M×N images if the collage includes at least Mrows and N columns, wherein M and N are positive integers.

The style and the format of the image product can have defaultselections. For example, a photobooks can have “everyday” and hardcoverfor a book style, and 8″ by 8″ for the photobook format. In someembodiments, the style and the format of the photobook can be selectedbased on knowledge of the images in the group. Specifically, an imageproperty can be extracted from the group of identified images. Forexample, if the images are identified by a common tag label “HawaiiVacation”, a photobook style for vacation, or specifically for Hawaiianvacation, may be automatically selected. In another example, if theimages identified are in an electronic album called “Molly's BirthdayParty”, a birthday photobook style can be automatically selected. Inanother example, the album name 315 (FIG. 3) in which the images (320,FIG. 3) are identified can be used to automatically select a photobookstyle (such as a “Graduation” photobook style).

The computer system automatically produces a design incorporating theidentified images using the automatically selected style and format forthe image product (step 240). It should be noted that the design of theimage product using the group of identified images, and the associatedselections of style and formats, are proactively and automaticallyimplemented by the computer system without receiving a request from auser. The approach described in the present systems and methods isdifferent from conventional processes for crating image products whichrequire users to initiate projects to design the image products. Asdescribed above, the conventional methods require awareness andfamiliarity of the image products by the users. The projects of creatingimage products such as photobooks can be very time consuming. A lot oftime must be committed by a user without her knowing how an imageproduct could look like at the end.

The proactively and automatically created design of the image productsuch as a photobook can be presented by the computer system to a userfor preview (step 250). For example, as shown in FIG. 4, the userinterface 300 can display a button 400 “See Your Photobook” to indicatethat a photobook based on the images 320 in the image album 310 has beenproactively created, and ready for the users view. A photobook design500 is displayed in the user interface 300 after the user clicks thebutton 400, as shown in FIGS. 5A and 5B. The photobook can include afront cover 510, a back cover 520, and a plurality of pages 530, 540.The book title can be automatically selected from the name of the imagealbum 310 (FIG. 3). The front cover 510 can include a book title 515 andan image 518 selected from the images 320 (FIG. 3). For example, theimage 518 can be the first image in the album 310 or the image used (fora thumbnail image) to represent the image album 310. The page 530 caninclude an image 531. The page 540 can include images 541, 542, and text543. The presentation of the photobook design 500 can include realisticillumination and texture to imitate effects of the materials in theselected book format.

The user can click arrow buttons 550, 560 to view different pages of thephotobook. The cover and different pages of the photobook design 500 canalso be played automatically like a slide show once the user clicks thebutton 400 (FIG. 4).

Optionally, the user is allowed to edit the design of the image productsuch as the photobook design 500 (step 260). The user can change,switch, or remove the images 518, 531, 541, and 542 on the book cover510 and pages 530, 540. The user can also change text information suchas the book cover 515 and the text 543. It should be noted that thephotobook design 500 represents a complete design of a photobook that isready to be ordered. The editing is an optional step depending on theuser's time availability.

It should be understood that the proactively created designs of theimage product can be presented in many forms. As describe above, imagesidentified to create a photobook design typically belong to a group,such as a group of shared images, posted images, uploaded images, orimages selected for printing, etc. The proactively created photobookdesign is presented to the user at relevant places on the web pages,such as on a web page for confirming the share of the group of images,at a share web when the group of shared images are viewed, or at a pageafter a group of images have been ordered for printing. The imageservice provider can also proactively send an email to inform someoneabout the creation of the photobook design. For example, a user thatjust uploaded a group of images to the server managed by the imageservice provider may receive an email message from the service providerabout the creation of the photobook design.

After the user has reviewed and optionally edited the photobook design500, the user can save the design of the image product (e.g. thephotobook design 500) and order a physical manifestation of the imageproduct (e.g. the photobook) based on the to edit the design of theimage product (e.g. the photobook design 500) from the image serviceprovider (step 270).

The image service provider can receive the order via the server 160 andmanufacture a physical manifestation of the image product at theprinting facility 170 according to the design of the image product (e.g.the photobook design 500) (step 280).

Details about the proactive design of a photobook (in steps 230-240,FIG. 2) are described below. Referring to FIG. 6, a group of imageshaving numbers within a predetermined range are first obtained asdescribed above by a computer system (step 600). A style and a formatcan be automatically selected for a photobook design (step 610). Theimages are sorted by one or more criteria to produce a desirablesequence that the images will be presented in a photo book (step 620).The criteria can include a primary criterion. For example, the imagescan be sorted by their associated image capture times. The image capturetimes can be extracted from the header information in the image files.For example, a JPEG image may include an EXIF header that includes thecapture time of the image. It is stored by the digital camera in theimage file as information in association with the image data. Referringto FIG. 7, a cumulative image count can be plotted against image capturetime. In one implementation, the list of images can be sequencedchronically based on capture time of the images. Users often prefer tohave images to appear in book pages in a chronological order. The imagescan be then separated into page groups using one or more secondarycriteria. For example, images can be grouped according to capture time.Images captured in a short time interval are more likely suitable to beplaced on the same page, facing pages, or adjacent pages in a photobook. In some embodiments, images that are taken in a short interval(e.g. a day) can be assigned to a same group (e.g. Group A, FIG. 7). Theimages in Group A, Group B, or Group C (FIG. 7) are captured in a shorttime interval. The images in each of these groups can be respectivelygrouped as candidates to appear on a same page, facing pages, oradjacent pages.

The sorted images may be first placed into a group including a largenumber of images that do not fit on a page. For example, ten images inGroup B (FIG. 7) are taken in a short time interval such a day. Theimages in Group B can be further separated into page groups using othersecondary criteria such as color content and image capture locations.For instance, the most popular color of each of the ten images can beplotted in a color space (such as (Y, u, v), (L, a, b) etc.), as shownin FIG. 8. The color space is first separated into finite number ofregions. The most popular color is defined as the color region that hasthe most pixel values that fall into. The ten images can be furthersub-grouped according to their adjacencies in the color plane (u, v):images 1-3 may be categorized into a first group; images 4-6 may becategorized into a second group; images 7-10 may be categorized into athird group. The images 1-10 in Group A might have been taken within asame day. The photographer might have traveled to several differentsurroundings, such as a beach, a restaurant indoor, or a swimming poolat the hotel, each of which may have different color distributions. Itshould be understood that the display of color content distributions ofimages are not limited to a specific color coordinates or the twodimensional color plane shown in FIG. 8. The color distributions can beplotted in three dimensional color spaces (Y, u, v), (L, a, b), or twodimensional color planes such as (Y, u), (Y, v), (a, b), (L, a) etc. Insome embodiments, the images with close distance in color space areintentionally separated across different pages in the photobook such asthe images on a page will not look alike.

In some embodiments, the images can be sorted or grouped based ongeographic locations. For example, assuming Group C (FIG. 7) includesfive images that are taken, for example, in a same day. Geo locationinformation about each of the five images can be extracted from theheader information of the five images, and plotted as shown in FIG. 9.Images 1-3 are shown to have been taken in close proximity while images4-5 are taken at similar locations. Images 1-3 can be sub-groupedtogether and assigned to Page 4 of the photo book. Images 4-5 can besub-grouped together and assigned to Page 5 of the photo book. In someembodiments, the Images 1-3 and the Images 4-5 can be assigned to be onfacing pages (e.g. such as pages 4 and 5) such that the images from thesame events can be viewed on facing pages in one open-book view. In someembodiments, the images with close geographic proximity areintentionally separated across different pages in the photobook such asthe images on a page will not look alike. The placements of two pagegroups of images on facing pages can be based on one or more secondarycriteria, such as image capture time, image color content, and imagecapture location.

A page layout is automatically selected from the library of page layoutsin accordance with the properties of the images on the page as well asthe text information associated with the image receiving areas in thepage layout (step 630). The format of the image receiving field in thepage layout matches the format of the selected image. For example,images in Group A (FIG. 7) may include two landscape images (L) and twoportrait images (P). A notation for the image formats for Group A can be“LLPP”. The page layout selected from the page-layout library shouldhave four image receiving areas with matching image formats “LLPP”.

The page layout also includes an image caption area positioned adjacentto the image receiving area for receiving the associated textinformation. For example, one portrait image and one landscape image maybe selected for the next page. The portrait image has associated textinformation entered by the user and stored in conjunction with theportrait image. The landscape is not associated with any textinformation. In selecting a page layout, the disclosed system firstselect page layouts containing two image receiving areas, and thenfurther selects page layouts that have include one portrait and onelandscape image receiving areas. Additionally, the disclosed systemfurther selects page layouts that include an image caption area inassociation with the portrait image receiving area. After the pagelayout is selected, the selected image(s) and associated text areautomatically placed in the selected page layout (step 640). Theplacement of the selected image(s) and associated text can be done in asingle step because they are pre-stored in an integrated data structureas described previously. The same steps can be repeated for fill theimage fields and text fields in each page of a photo book. The user canselect an image from the plurality of images to be used as part of thebook cover. Details about arranging images in page layouts are disclosedin the above referenced commonly assigned U.S. patent application Ser.No. 12/406,873, titled “Smart photobook creation”, the content of whichis incorporated herein by reference.

In some embodiments, the design of an image product can be dynamicallychanged as the identified images change in the group. For example, thedesign of the image product can dynamically change as a user adds orremoves images in and out of an image album (310, FIG. 3). The number ofpages, the number of images per page, and the layout of images on thepages in a photobook, for example, can automatically adapt to thechanges to the images identified in the group. In general, the processof updating the design of an image product can include the followingsteps: similar to the descriptions above in relation to steps 210-250(FIG. 2), referring to FIG. 10, a group of images is first identified(step 1010). If the number of images in the identified group of imagesis determined to be in a predetermined range (step 1020), a style and aformat are automatically selected for the design of an image product(step 1030). A first design of an image product is automatically createdwithout receiving a request from a user (step 1040). The first design ofthe image product is made available for the preview by a user (step1050). When a change is detected in the group of identified images (step1060), a second design of the image product is automatically createdwithout receiving a request from a user in response to the changedetected in the group of images (step 1070). The change in the group ofimages can include addition to the group of images or removal of imagesfrom the group of images. The change in the group of images can alsoinclude a change in an image property of the group of images. The imageproperty can include a tag, a keyword, a title of a folder where theimages are placed, or metadata at least one of the images.

The second design of the image product is subsequently presented foruser's preview (step 1080). The change in the group of identified imagescan include addition, removal, and a change in the sequence, addition ofremarks and keywords, addition in change in tagging in a group ofimages. The uploading of images into an image folder/album can take morethan one uploading events. The design of the image product can becontinually updated as new images are received by the imagefolder/album.

In some embodiments, referring to FIG. 11, a library of specificationterms for image products is stored in a computer system (step 1110). Thecomputer system can include a network based computer system, a userdevice, a combination thereof, or in other types of configuration. Thespecification terms cover all necessary information for creating animage product. For example, the specification terms include the names ofimage products or image services, or the types of image products andservices, such as photobook, greeting card, calendar, photo mugs, photoT-shirts, photo aprons, single photo pages, photo collage pages, a photostationery, photo banners, photo mugs, photo mouse pads, photokey-chains, photo collectors, photo coasters, “photostory” or “photostory”, phone case, and various photo gifts and photo novelty items,etc. The library can store variations of the specification terms. Forexample, a photobook can also be referred to as a scrapbook or a book. Agreeting card can have different variations such as photo greeting card,holiday card, Christmas card, birthday card, Valentine card, invitationcard, thank you card, announcement card, get well card, etc. Otherexamples of image products and services are available at websitesoperated by Shutterfly™, Tinyprints™, MyPublisher™, ThisLife™, etc.

The specification terms can also include contact names or equivalentassociated with a user or in a user's address book, which can be storedin the user's computer device or at a central location operated by aservice provider. The contact names can include family members, friends,and acquaintances. For example, the person grandma is defined in contextof the user. The contact names can also include faces recognized in theuser's images stored in the user device(s), at an image service providersuch as Shutterfly™, Inc., or at a third party service provider such asFacebook™, Instagram™, etc. In relation to image products, the contactnames can be the subjects in the photos to be incorporated in an imageproduct, or the recipients for whom the image product is made.

The specification terms can include time period within which the imagesare to be selected from. A photobook for the year 2013 usuallyincorporates images taken in 2013. Unless specified, a holiday cardrefers to the current holiday season or the holiday season coming up.The specification terms can include occasions for which an image productis to be created for: for example, Christmas, wedding, birthday, soccergame, anniversary, vacation, trip to the Caribbean's.

The specification terms can include product styles such as modern,romantic, tropical, or styles developed by designer. The specificationterms can include color preferences such as warm, cool, blue, red,green, etc.

The specification terms can define the sources where the images will beobtained from. For example, the image sources can include user devicessuch as a phone, a user desktop computer, or a back drive in user'shome, image service provider such as Shutterfly™, third party servicessuch as Facebook™, Instagram™, Pinterest™, and cloud storage servicessuch as Amazon™, Google™, Dropbox™, Carbonite™, etc.

The specification terms can include one or more action words in a usercommand for creating an image product. Examples of action words include“make”, “create”, “design”, “form”, “develop”, “produce”, “order”,“buy”, etc. These action words are often used at the beginning of a usercommand, and can be used to clearly identify a user command for creatingan image product.

A command is received from a user (step 1115) by the computer system.The command includes a text formed by a string of characters, which canbe input by a user, or recognized from a voice message of a user. Theuser command is typically in a natural language comprising words in astring of characters. Examples for a user command include “Create aphotobook for 2013”, or “Make a 2014 photo calendar for grandma”, or“Make a photobook for Peter's soccer team”.

A lexical analysis is conducted on the command (step 1120) by thecomputer system. The lexical analysis can include parsing or syntacticanalysis of the text in the user command. The text string in the usercommand is tokenized into a plurality of tokens (step 1125).

The tokens obtained from the user command are then matched by thecomputer system to the specification terms in the library to determinespecification parameters for an image product (step 1130). The step ofmatching can include first converting the tokens to the standard formsto be consistent to the specification terms stored in the library, whichcan include removing the article the” or “a” in the front of a token,simplifying plural to a singular form, identifying synonyms ormorphological variations of a standard term and converting them tostandard forms. The step of matching then involves matching the tokensin their standard forms to the specification terms in the library.

The matched specification terms can determine specification parametersfor an image product. For example, tokens “photobook” and “2013” in theuser command “Create a photobook for 2013” define a photobook thatinclude photos taken in the year 2013. These action words such as “make”or “create” can be used to identify a user command for creating an imageproduct.

Optionally, the user can be asked by question(s) by the computer systemto define additional specification parameters for the image product(step 1135). For example, in the user command “Create a photobook for2013”, the additional question can include “For whom the photobook willbe made?” or “Is the photobook for the family?” (meaning “will onlyphotos of family members be included?”). Some messages like “Create aphotobook” or “Make a birthday card” do not provide sufficientinformation to specify an image product. The computer system can promptquestion to the user to clarify information such as the time period, thesubject in the photos of the photobook, whose birthday the card is for,from whom, etc. User's answers to these additional questions areanalyzed similar to the original user command in steps 1115-1130 above.

An image product type is automatically determined by the computer systembased on the matched specification parameters that correspond to aproduct type such as “photobook” or “birthday card” in the library (step1140).

Additionally, a style or a format for the image product can beautomatically selected based on the specification parameters (step1145). For example, if the user command or user answers to questionsinclude “8 by 8 inch photobook”, “wedding book”, “folded greeting card”,or “two piece iPhone 5s case”, an 8″×8″ photobook, a premium weddingstyle photobook, a 5″×7″ folded greeting card, and a two piece phonecase for iPhone 5s can be respectively selected automatically.

Images are automatically identified and selected based on thespecification parameters (step 1150), which can include severalsub-steps.

First, images may be evaluated in a subset of or all available imagesources. If a user command such as “Create a photobook for 2013” doesnot specify the sources of the images the image product is toincorporate, the images satisfying the criteria (e.g. taken in 2013)from all available image sources can be evaluated. Alternatively, theuser command or answer may include image source specifications such as“Create a photobook for 2013 using pictures in my Shutterfly account”,or “Create a photobook for 2013 with photos from Shutterfly, Facebook,and Instagram”. Images will be retrieved and evaluated in the specifiedimage sources. As described above, image sources can include the imagesources can include user devices such as a phone, a user desktopcomputer, or a back drive in user's home, image service provider such asShutterfly™, third party services such as Facebook™, Instagram™,Pinterest™, and cloud storage services such as Amazon™, Google™,Dropbox™, Carbonite™, etc.

Secondly, the automatic identification and selection of images can bebased on the person(s) and the subject defined in the specificationparameters (i.e. matched specification terms). For example, if the imageproduct to be made is a family photobook or to be given to Grandma, onlyphotos of family members (potentially with emphases on grand children)will be first selected. The person(s) in the photos can be identifiedusing tags or keywords associated with the photos or faces recognized inthe photos. In another example, if a user command recites “a photobookfor Hawaii vacation”, then the photos taken in Hawaii will be firstselected for evaluation. The photo capture location or vacation can bedetermined using geo location tag, keywords, and album names inassociation with the photos.

Next, the images identified in the (designated) image sources are rankedbased on predetermined criteria such as image quality, relevance,colors, orientation, format, and similarity among images, etc. Forexample, a photo that is fuzzy or of low resolution has lower imagequality. The ranking can produce ranking values for the images. Detailsfor ranking images for image products and services are also disclosed incommonly assigned U.S. Pat. No. 7,467,222, the disclosure of which isincorporated herein by reference.

The images are then selected based on the ranking Higher ranked imagesare selected. Lower quality and less relevant images are discarded.Similar or redundant photos can be trimmed. Similar to what is describedpreviously in relation to FIG. 2 (step 220) and FIG. 10 (step 1020), thenumber of images selected is dependent on the type of image product tobe created. In other words, a subset of the identified images areselected comprising an image number within a predetermined rangecompatible with the image product type For example, a 12 month bookcalendar typically includes a cover photo, and 1-5 photos per month sothe total number of images can be limited to less than 60, such as 30. Aphotobook on the hand should at least include 20 images. A Christmascard typically includes less than 6 photos, and most commonly, only onephoto.

With product type, format, style and image selected, a design for theimage product is automatically created using the selected images and thestyle or format for the image product type (step 1155). The layout ofthe images in the image product can be determined capture times, geolocations, image properties (color, density, sharpness, tone scale,image quality, etc.). Examples of automatic image layout in imageproducts are described above in association with FIGS. 7-9.

The design of the image product can be presented to a user for preview(step 1160). The presentation can be made at a web user interface suchas in a web browser, in a mobile application, or in a design file, etc.The user can be allowed to edit and enter additional information such asbook title and captions within the pages. The user can decide to orderan image product based the image product design, or share the imageproduct design.

The recipient for which the image product is created can be defined bythe specification parameters obtained in the step of matching (step1130). For example, after being manufactured, a photobook made forgrandma is shipped to the grandma (the role of grandma is defined in thecontext of the user).

An advantage of the above disclosed methods is that the user does notneed to go through a length process for creating a personalized imageproduct as in the convention systems. Another advantage of the abovedisclosed methods is that an image product is automatically created byunderstanding a user command. The user does not need to go over multiplesteps to select image product type, product format, and product style,or select images to be placed in the image product. The disclosedmethods can take user command in a natural language in text or voice andthus are more convenient and can save time for users.

It should be understood that the presently disclosed systems and methodsare suitable for proactively creating different image products otherthan photobooks. The image products can include multiple pages or viewsthat each can display one or more images, such as, photo books, photocalendars, photo scrapbooks, photo snapbooks, photo calendars, photoalbums, and a multi-face photo card. The image products can also includea single page or view for displaying one or more images, which caninclude, for example, photo greeting cards, holiday cards, photo mugs,photo T-shirts, photo aprons, single photo pages, photo collage pages, aphoto stationery, photo banners, photo mugs, photo mouse pads, photokey-chains, photo collectors, and photo coasters etc.

Furthermore, it should be understood that the library of page layoutcontains layouts with different numbers of images other than one, two,and three. For a given number of images on a page and for the givenformats of the images, the page layout is not limited to the designsshown in this application. There can be a large variation of layoutdesigns for the same number of images with the same formats on eachpage. Moreover, the sorting and grouping of images can be implemented byother approaches or using other metrics. The illustration of pagegrouping and page breaks for the images can also vary within the scopeof the present disclosure. Moreover, the presently disclosed systems andmethods are compatible with a software application installed locally ona computer device, client software in communication with a remoteserver, or a software application installed on a central server.

What is claimed is:
 1. A method for proactively creating an imageproduct, comprising: storing a library of specification terms for imageproducts by a computer system; receiving a command from a user;tokenizing the command into a plurality of tokens by the computersystem; matching one or more of the tokens to the specification terms inthe library to determine specification parameters for an image productby the computer system; automatically identifying images based on thespecification parameters by the computer system; and automaticallycreating, by the computer system, a design for the image product thatincorporates at least some of the images identified based on thespecification parameters.
 2. The method of claim 1, wherein thespecification terms in the library specify one or more image sourcesfrom which the images are to be obtained, wherein one or more of thetokens are matched to the specification terms in the library thatspecify the one or more image sources, wherein the step of automaticallyidentifying comprises identifying images at the one or more imagesources.
 3. The method of claim 1, wherein the specification terms inthe library specify one or more persons, wherein one or more of thetokens are matched to the specification terms in the library thatspecify one or more persons, wherein the step of automaticallyidentifying comprises identifying images that include the one or morepersons in the images.
 4. The method of claim 1, wherein thespecification terms in the library specify an occasion that the imagesare associated with, wherein one or more of the tokens are matched tothe specification terms in the library that specify an occasion, whereinthe step of automatically identifying comprises identifying images thatare associated with the occasion.
 5. The method of claim 1, wherein thespecification terms in the library specify a time period in which theimages are taken, wherein one or more of the tokens are matched to oneor more specification terms in the library that specify a time period inwhich the images are taken, wherein the step of automaticallyidentifying comprises identifying images that are taken in the timeperiod.
 6. The method of claim 1, wherein the specification terms in thelibrary specify one or more image product types, wherein one or more ofthe tokens are matched to at least one specification term in the librarythat specifies an image product type, wherein the design for the imageproduct is automatically created in the image product type.
 7. Themethod of claim 1, further comprising: selecting a subset of theidentified images, wherein a number of the identified images in thesubset is within a predetermined range compatible with the image producttype.
 8. The method of claim 1, wherein the specification terms in thelibrary specify a style or a format for an image product, wherein one ormore of the tokens are matched to at least one specification term in thelibrary that specifies a style or a format for an image product, whereinthe design for the image product is automatically created in the styleor the format.
 9. The method of claim 1, further comprising:manufacturing a physical manifestation of the image product based on thedesign for the image product.
 10. The method of claim 9, wherein thespecification terms in the library specify a recipient for an imageproduct, wherein one or more of the tokens are matched to aspecification term in the library that specifies a recipient for animage product, wherein the physical manifestation of the image productis sent to the recipient after the step of manufacturing.
 11. The methodof claim 1, further comprising: ranking the identified images usingpredetermined criteria; and selecting a subset of the identified imagesbased on a result of the ranking step, wherein the design for the imageproduct incorporates the subset of the identified images.
 12. The methodof claim 1, further comprising: after the step of matching, presenting aquestion to the user by the computer system; receiving an answer fromthe user; tokenizing the answer into a plurality of tokens by thecomputer system; and matching one or more of the tokens to thespecification terms in the library to determine additional specificationparameters for an image product by the computer system.
 13. The methodof claim 1, wherein the step of automatically creating comprises:assigning at least a portion of the identified images into a first pagegroup and a second page group each comprising one or more images;placing the first page group of one or more images in the first page ofthe image product; and placing the second page group of one or moreimages in the second page of the image product.
 14. The method of claim1, wherein the step of receiving a command comprises: receiving a textfrom the user, wherein the text is tokenized by the computer system. 15.The method of claim 1, wherein the step of receiving a commandcomprises: receiving a voice message from the user; and recognizing atext in the voice message, wherein the text is tokenized by the computersystem.
 16. The method of claim 1, wherein the step of tokenizingcomprises parsing the command.
 17. The method of claim 1, wherein thestep of matching comprises matching the one or more of the tokens tomorphological variations of the specification terms in the library todetermine specification parameters for an image product by the computersystem.